The Use of Visible Speech Cues (speechreading) for Directing Auditory Attention: Reducing Temporal and Spectral Uncertainty in Auditory Detection of Spoken Sentences

Classic accounts of the benefits of speechreading to speech recognition treat auditory and visual channels as independent sources of information that are integrated early in the speech perception process, most likely at a pre-categorical stage . The question addressed in this study was whether visible movements of the speech articulators could be used to improve the detection of speech in noise, thus demonstrating an influence of speechreading on the processing of low-level auditory cues. Nine normal-hearing subjects detected the presence of spoken sentences in noise under three conditions: auditory-only (A), auditory-visual with a visually matched sentence (AVM), and auditory-visual with a visually unmatched sentence(AVUM). When the video matched the target sentence, detection thresholds improved by about 1.5 dB relative to the auditory-only and auditory-visual unmatched conditions. The amount of threshold reduction varied significantly across target sentences (from 0.73 to 2.19 dB). Analysis of correlations within each sentence between area of mouth opening and four acoustic envelopes (wideband, F1 region, F2 region, and F3 region) suggested that AVM threshold reduction is conditioned by the degree of auditory-visual spectral comodulation.